Deliverable D 3 . 10 Working Paper on Dialogue Management Evaluation April 1999 Esprit Long - Term Research Concerted

This DISC Working Paper present a first outline of a systematic approach to the evaluation of spoken language dialogue systems (SLDSs) and their components. The approach is illustrated for the case of dialogue manager evaluation. The basic components of the approach are (a) a model of the core properties of the SLDS or component to be evaluated; (b) a set of evaluation criteria which have been systematically generated from the model; (c) a generic template for characterising individual evaluation criteria with respect to, among other things, which core property is being evaluated, the types of evaluation involved, which methods to use, when to evaluate the property during the life-cycle, importance of evaluation, difficulty and cost of evaluation; and (d) filled templates for each individual evaluation criterion. Using this approach, 35 evaluation criteria for dialogue manager evaluation are described.